The present invention is directed to an improvement in distributed systems and in particular to remote procedure call messages between multi-threaded processes in a distributed system.
Distributed computer systems involve different processes which may execute simultaneously. The processes may be found on physically distinct processors or the processes may be implemented on a single processor. In multi-threaded distributed systems, each process may run one or more threads. Such distributed multi-threaded, multi-process systems are well known, particularly in real time processing applications such as systems for telephone control and management. Multi-threaded, multi-process distributed systems raise timing, scheduling, queuing, synchronizing and interprocess/interprocessor communications issues which must be solved to ensure that the distributed system runs efficiently and effectively.
In multi-threaded, multi-process distributed systems, one characteristic function made available in the system is a remote procedure call (RPC) function. An RPC permits a process in the distributed system to request that a different process in the system carry out a function implemented in that second process and, return information to the process having originated the RPC request. The RPC is implemented by an RPC message being transmitted across the distributed system such that the responding process receives the RPC and takes the appropriate steps as defined by the RPC message.
In distributed systems implemented using a TCP/IP protocol, the RPC will take the form of a TCP/IP-compliant message sent in accordance with the TCP/IP protocol.
In prior art multi-processor, multi-threaded systems, a thread which originates an RPC will open a communication channel (in the TCP/IP protocol, a socket) which will permit the thread in the process to communicate the RPC to a second process. Should there be several threads which originate RPCs, a corresponding number of channels, (or sockets) will be opened to permit each thread to communicate with the appropriate process. In such systems the channel will remain open until the response to the RPC is received by the originating thread, or the channel will be opened and closed for the transmittal of the RPC, and for the receipt of the RPC, incurring the associated operating system overhead cost.
In such prior art systems, it is typical that the thread and process originating an RPC request will make use of a central registry (or broker) which provides the identity of the process able to respond to the RPC to the process originating the RPC. A direct connection is then established between the originating process and the responding process to permit the RPC request message to be communicated from the originating process to the responding process and the response to likewise be communicated.
In a distributed system where the numbers of channels (or sockets) is a constrained resource, it will be appreciated that the use of a communication channel (or socket) by each thread wishing to originate an RPC, will create resource allocation, or operating system overhead, problems for the distributed system. In addition, the use of a central broker or registry creates overhead problems for such a distributed system.
It is therefore desirable to have an improved distributed system in which it is not necessary for each thread in the multi-threaded multiprocessing distributed system to request allocation of a channel (or socket) to permit the transmittal of an RPC message. Similarly, it is desirable to have a system which does not rely upon a central broker or registry of processes to be used by the system to determine which processes are appropriate to receive a given RPC message.
According to one aspect of the present invention, there is provided an improved multi-threaded, multi-process distributed system.
According to a further aspect of the present invention, there is provided an improved multi-threaded, multi-process distributed system comprising a plurality of processes, each of the said processes comprising a plurality of threads and a means for receiving, composing, and forwarding remote procedure call messages, the distributed system comprising, a queue associated with each of the threads, each of the queues in each of the processes being associated with a predetermined queue identifier selected to uniquely identify each queue within each process, a set of connections selectively established between the processes, each one of the set of connections being associated with a predetermined connection identifier selected to uniquely identify the connection with the distributed processing system, the remote procedure call messages each comprising, a first data element corresponding to a selected one of the queue identifiers, and a second data element corresponding to a selected one of the connection identifiers.
According to a further aspect of the present invention, the improved multi-threaded, multi-process distributed system as set out above is provided in which the set of connections consists of connections characterized by each pair of processes selected from the processes in the distributed processing system having no more than one connection between the pair of processes at any one time.
According to a further aspect of the present invention, the improved multi-threaded, multi-process distributed system as set out above is provided in which a one of the remote procedure call messages, having been forwarded over a connection by a first one of the processes, to a second one of the processes, for forwarding to a third one of the processes, is characterized by, its first data element corresponding to the queue identifier of a one of the queues in the second one of the processes, and its second data element corresponding to the connection identifier of the connection from the first one of the process to the second one of the processes, and wherein the one of the remote procedure call messages, when composed by an originating one of the processes, is characterized by its first data element corresponding to the queue identifier of an originating one of the queues in the originating one of the processes, and its second data element being set to a predefined null value.
According to a further aspect of the present invention, the improved multi-threaded, multi-process distributed system as set out above is provided in which in each process the means for receiving, composing, and forwarding remote procedure call messages comprises update means for modifying the second data element in a received one of the remote procedure call messages by storing, in the second data element, the connection identifier of the connection by which the received one of the remote procedure call messages was forwarded to the process, means for determining whether the received one of the remote procedure call messages is processable within the process, means for processing the received one of the remote procedure call messages when the selected one of the remote procedure call messages is determined to be processable within the process, means for composing a remote procedure call message response when the selected one of the remote procedure call messages has been processed within the process, means for forwarding the remote procedure call message response by the connection indicated by the second data element of the received one of the remote procedure call messages, means for selecting, from the set of the processes of the distributed processing system, when it is determined that the received one of the remote procedure call messages is not processable within the process, a further different process for processing the received one of the remote procedure call messages, and means for forwarding the received one of the remote procedure call messages to the further different process, means for storing in a data location local to the process, the first and second data elements of the received one of the remote procedure call messages, means for updating the first data element of the received one of the remote procedure call messages to identify a one of the queues in the process.
According to a further aspect of the present invention, there is provided a method for creating and forwarding remote procedure call messages in a distributed processing system, the distributed processing system comprising a plurality of processes, each of the said processes comprising a plurality of threads and a means for receiving, composing, and forwarding remote procedure call messages, a queue associated with each of the threads, each of the queues in each of the processes being associated with a predetermined queue identifier selected to uniquely identify each queue within each process, a set of connections selectively established between the processes, each one of the set of connections being associated with a predetermined connection identifier selected to uniquely identify the connection with the distributed processing system, the method comprising the steps of
a) selectively establishing a set of connections between the processes in the distributed processing system, whereby a first selected process is connected to a second selected process by no more than one connection, and
b) forwarding one or more remote procedure call messages in the distributed processing system by storing, updating and referencing queue identifier and connection identifier data in the remote procedure call messages.
According to a further aspect of the present invention, the method as set out above including the steps of:
a) responding to an originating remote procedure call in a first thread in a first process by creating a remote procedure call message in the first process comprising
i) a first data element corresponding to the queue identifier of the first thread, and
ii) a second data element being set to a predefined null value.
b) selecting a second process for receipt of the remote procedure call message,
c) forwarding the remote procedure call message to the second process,
d) in the second process, modifying the second data element to correspond to the connection identifier of the connection between the first process and the second process,
e) in the second process, determining whether the remote procedure call message is processable within the second process,
f) when the remote procedure call message is determined to be processable within the process:
g) processing the remote procedure call message within the process,
h) creating a response in the remote procedure call message,
i) returning the remote procedure call message by the connection indicated by the second data element,
j) when it is determined that the received one of the remote procedure call messages is not processable within the process:
i) selecting, from the set of the processes of the distributed processing system, a third process for processing the remote procedure call message,
ii) storing in a data location local to the second process, the first and second data elements of the remote procedure call message,
iii) updating the first data element to identify a one of the queues in the second process, and
iv) forwarding the remote procedure call messages to the third process.
Advantages of the present invention include a multi-threaded, multi-process distributed system in which a single communication channel between processes may be utilized to communicate RPC messages from different threads within the same process. Further advantages include a multi-threaded, multi-process distributed system which maintains the information relating to which process is appropriate to respond to a given RPC, within the originating process, rather than at a central broker or registry.